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DESCRIPTION 

LATENCY HANDLING FOR INTERCONNECTED DEVICES 

5 The present invention relates to systems composed of a plurality of 

devices clustered for the exchange of audio and/or video data and control 
messages via wired or wireless link and, in particular although not essentially, 
to such systems where different data components from a source device are to 
be routed to respective and separate other devices of the system. The 
10 invention further relates to devices for use in such systems. 

Networking or interconnection of devices has long been known and 
used, starting from basic systems where different system functions have been 
provided by separate units, for example hi-fi or so-called home cinema 

1 5 systems. A development has been the so-called home bus systems where a 
greater variety of products have been linked with a view to providing enhanced 
overall functionality in for example domestic audio/video apparatus coupled 
with a home security system and the use of telephone. An example of such a 
home bus system is the domestic digital bus (D2B), the communications 

20 protocols for which have been issued as standard IEC 1030 by the 
International Electrotechnical Commission in Geneva, Switzerland. The D2B 
system provides a single wire control bus to which all devices are interfaced 
with messages carried between the various devices of the system in a 
standardised form of data packet. 

25 A particular problem that can occur with distributed systems such as hi- 

fi and home cinema is loss of synchronisation between different components 
required to be presented to a user simultaneously, in particular for video 
images and an accompanying soundtrack, or between different components of 
the soundtrack, where the different components are to be handled by different 

30 devices - for example in a home cinema set-up. This loss of synchronisation 
may occur due differences in the effective lengths of the transmission paths 
for the differing components resulting in, or due to, different latencies in 
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decoders or intermediate processing stages for the different components. 

One way to approach the synchronisation problem, where all the 
components are decoded within a single device, is described in US 5,430,485 
(Lankford et al) which describes a receiver for decoding associated 
compressed video and audio information components transmitted in mutually 
exclusive frames of data, each with a respective presentation time stamp. A 
coarse synchronisation is applied by selectively dropping frames of one or 
other of the components and then fine tuning by adjusting the audio stream 
clock frequency. 

Another approach, this time closer to the source for different 
components being sent out, is described in US 5,594,660 (Sung et al) which 
provides an audio/video decoder/decompressor for receiving and separating 
the components of an encoded and compressed data stream. Within the 
decoder/decompressor, Sung has means for breaking up a compound AV 
stream and then applying an appropriate temporal offset to each stream to 
achieve synchronisation of the outputs during playback. The differential 
buffering by FIFO units follows the system decoder but precedes the decoding 
of the audio or of the video. 

Although handling the component delays with the components still 
encoded generally involves less processing, handling of synchronisation 
(particularly if done at source) can create its own problems when it comes to 
determining just how much delay is to be applied to each component stream. 

It is accordingly an object of the present invention to provide a 
networked system of devices including enabling means for synchronising 
components intended to be presented synchronously to a user of the system. 

In accordance with the present invention there is provided a data 
processing system comprising a cluster of devices interconnected for the 
communication of data in streams wherein, for at least two data streams to be 
sent to one or more devices as destination devices of said cluster, at least 
device of the cluster comprising buffering means arranged to apply a respective 
delay to at least one of said at least two data streams in an amount determined 




3 PHGB 000005 

by differing signal path latencies for said at least two streams; 

characterised in that at least some devices of the cluster maintain a 
respective table, readable via said interconnection by other devices of said 
cluster, each such table identifying one or more latencies for the respective 
5 device, and the buffering means applying delays on the basis of table contents. 
By the use of respective tables, which are suitably (but not essentially) carried 
by all destination devices, the determination of what delay to apply to each data 
stream made be made more simply and greater flexibility is introduced to the 
system in that changes to processing arrangements may just require a table 

io entry to be altered, rather than a large-scale revision of the recorded operational 
parameters typically held by networked devices. 

Each table may identify, for its respective device, signal processing 
capabilities for that device, together with the latency associated with each such 
capability. Where one of the devices is a source device for said at least two 

15 data streams to be sent to said destination devices of said cluster, said source 
device may include the buffering means together with means arranged to read 
data from said respective tables of the destination devices and determine the 
respective delay to apply to at least one of said at least two data streams. In 
such an arrangement, the source device may further comprise multiplexing 

20 means coupled with the buffering means and arranged to combine said at least 
two streams into a single data stream for transmission to said destination 
devices. 

Whilst simple figures for the respective delays may be held in each table, 
one or more table entries may be in the form of an algorithm requiring data from 

25 the device reading the table to enable determination of the latency of the device 
holding said table. For this, the determination on the basis of the algorithm may 
be implemented by the device reading the table, said device having downloaded 
the algorithm from the device holding the table: alternatively, the determination 
may be implemented by the device holding the table, with the results of the 

30 implementation being transmitted via said interconnection to the device reading 
the table. 

The present invention also provides a data processing apparatus 
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comprising the technical features of a source device in a system as recited 
hereinabove and as claimed in the claims attached hereto, to which the readers 
attention is now directed. 

5 Further features and advantages of the present invention will become 

apparent from reading of the description of preferred embodiments of the 
invention, given by way of example only and with reference to the 
accompanying drawings, in which: 

Figure 1 represents an arrangement of three interconnected devices 
10 forming an audio/video cluster; 

Figure 2 represents a table of latency information as carried by one of 
the devices in the cluster if Figure 1 ; 

Figure 3 represents a configuration of source device suitable to embody 
the present invention; and 
15 Figure 4 represents an alternative (wireless) interconnected cluster 

suitable to embody the present invention. 

A first arrangement of interconnected devices is shown in Figure 1 , with 
three devices 10, 12, 14 forming a cluster 16 based around a respective bus 

20 18 supporting communication in accordance with IEEE Standard 1394 
connect and communications protocols. In the following example, reference is 
made to IEEE 1394, and the disclosure of the specification of this protocol is 
incorporated herein by reference. As will be recognised by the skilled reader, 
however, conformance with such protocol is not essential to the operation of 

25 the present invention. 

The devices in the cluster 16 comprise a source device 10 coupled via 
bus 18 to a pair of presentation devices, in this example a television 12 for 
showing the image component of a combined AV stream from the source, and 
an audio processor and playback system 14 for reproducing the audio 

30 component of the AV stream. 

In order to synchronise the presentation to a user of the audio and video 
components, a device on the network must arrange for some stream 
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components (in this example the audio component) to be delayed relative to the 
others (in this case video). In the Figure 1 example, if a data stream from the 
source 10 to the two destination devices 12, 14 consists of MPEG2 video and 
AC3 audio, where the processing delay for MPEG2 in the television 12 is 1.0 
5 second, and the processing delay for AC 3 audio in the audio presentation 
device 14 is 0.1 seconds, the audio signal must be delayed by (1.0 - 0.1) = 0.9 
seconds at some point along its signal path to achieve synchronisation. One 
technique for applying this delay is described in our co-pending commonly- 
assigned application entitled "Interconnection of AudioA/ideo Devices" and will 
10 be briefly described hereinafter with respect to Figure 3. In order to be able to 
arrange for these delays, the system must have some means for determining 
the processing delays for the various types of data stream. 

In order to enable application of the appropriate delay to counter latency 
in a matching stream, particularly for those devices supporting more than one 
15 processing capability (e.g. MPEG2 and DV; AC3 and MP3), each device in the 
cluster 10, 12, 14 is provided with a respective internal or remotely stored look- 
up table 11, 13, 15. In this table (an example of which is given in Figure 2 for 
the television 12 from Fig.1) there will be one entry for each type of stream that 
a device can process. The entry will consist of, at least, an identifier for the type 
20 of stream, and a processing delay for that stream. Other information about the 
stream may be contained in the table, as required. 

In certain circumstances the system may support changes to the 
specified delays in response to user input varying one or other of the preset 
audio parameters. The notification for such changes will generally be in the form 
25 of a protocol-supported notification and the extent to which some or all devices 
of the cluster detect and record the effects of the change against their particular 
parameters as stored in their respective table will depend on the extent to which 
they follow the protocol. This also applies to their ability to read updated tables 
from other devices of the cluster. 
30 A single type of stream may have different processing delays if, for 

example, the device has different processing delays for various ranges of bit 
rates for that type of stream. Also, as shown by the entry for MPEG7 stream 
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types, an entry may consist of an algorithm to determine the delay. For 
example, if the delay was 0.1 seconds for every megabit per second of incoming 
data, the formula of (0.1 *x) seconds, where x is the number of megabits per 
second, could be stored. With an algorithm in the table, it is either packaged 
5 such as to be available for downloading by a device seeking to determine 
delays, or the enquiring device may be required to submit parameter values 
(e.g. x) to the device holding the algorithm in its table, which device would then 
calculate the delay and return the value to the enquiring device. The table may 
be accessed by some form of read transaction (e.g. "read 1 ' operations 

10 conforming to IEEE 1394 protocols), a command protocol (e.g. AV/C), a remote 
method invocation protocol (e.g. request messages according to the Home 
Audio/Video interoperability standard - HAVi - based around IEEE1394), various 
Java™ RMI procedures or some other method. 

As mentioned above, and described in greater detail in our co-pending 

15 application, one possible configuration for the source device 10 comprises an 
audio stream buffer 20 and a video stream buffer 22 for receiving separate input 
components from a remote signal source 24. Under the direction of a controlling 
processor 30, which reads the processing latencies from the tables (not shown) 
for destination devices 12, 14, the buffers are used to apply a respective delay 

20 to at least one of the two data streams to combat the differing processing 
latencies in the video 12 and audio 14 destination devices. Also under the 
direction of the processor 30, a multiplexer stage 32 combines the temporally 
offset audio and video from the respective buffers into a single data stream for 
transmission via the 1394 bus 18. 

25 Whilst the signals in the respective buffers 20, 22 may simply be read out 

and recombined, the source device optionally further comprises data processing 
means interposed in the signal path between the buffers 20, 22 and the 
multiplexer 32. As shown, this further data processing means may take the 
form of an audio signal processor ASP 34 on the output to the audio signal 

30 buffer and a video signal processor VSP 36 on the output to the video signal 
buffer. 

The first and second data streams (audio and video) may be encoded 
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according to a first communications protocol such as MPEG1 or 2, and the 
destination devices 12, 14 are each provided with a respective decoder 40, 42 
operating according to the said protocol. 

From reading the present disclosure, other modifications and variations 
5 will be apparent to persons skilled in the art, including equivalents and features 
which are already known in the field of bus-connected and cordless 
communication systems and components and which may be used instead of or 
in addition to features already disclosed herein. For example, as shown by 
Figure 4, the source 58 may comprise an optical or magnetic disk reader and, 

10 instead of a digital data bus, the data channel from source 60 to destination 
devices 62, 64, 66 may be a wireless communications link 68 for which each of 
the destination devices is provided with at least a receiver and the source device 
is provided with at least a transmitter. The system may comprise many more 
devices than illustrated herein including, for example, two or more source 

15 devices, and some devices of the system may have the technical features of 
both source and destination (for example a video cassette record and playback 
deck) with the appropriate source/destination behaviour being selected in 
dependence on the context. 

In the foregoing we have described a data processing system comprises 

20 a cluster of devices interconnected for the communication of data in streams, 
particularly digital audio and/or video data. One of the devices is a source 
device for at least two data streams to be sent to one or more other devices as 
destination devices of the cluster. To enable synchronisation of the stream 
presentations by the destination devices, some or all of the devices carry 

25 respective tables identifying, for that device, an identifier for each type of data 
stream that the device can process together with the processing delay for that 
stream. The or each such table is accessible via the cluster connection to 
whichever of the devices, at source, destination or in between for the signal, is 
handling application of the necessary offsets. 
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CLAIMS 

1. A data processing system comprising a cluster of devices 
5 interconnected for the communication of data in streams wherein, for at least 

two data streams to be sent to one or more devices as destination devices of 
said cluster, at least device of the cluster comprising buffering means arranged 
to apply a respective delay to at least one of said at least two data streams in an 
amount determined by differing signal path latencies for said at least two 
10 streams; 

characterised in that at least some devices of the cluster maintain a 
respective table, readable via said interconnection by other devices of said 
cluster, each such table identifying one or more latencies for the respective 
device, and the buffering means applying delays on the basis of table contents. 

15 

2. A system as claimed in Claim 1, wherein each table identifies, for 
its respective device, signal processing capabilities for that device, together with 
the latency associated with each such capability. 

3. A system as claimed in Claim 1 or Claim 2, wherein one of said 
devices is a source device for said at least two data streams to be sent to said 
destination devices of said cluster, said source device including said buffering 
means together with means arranged to read data from said respective tables of 
the destination devices and determine the respective delay to apply to at least 
one of said at least two data streams. 

4. A system as claimed in Claim 3, wherein said source device 
further comprises multiplexing means coupled with said buffering means and 
arranged to combine said at least two streams into a single data stream for 

30 transmission to said destination devices. 

5. A system as claimed in Claim 2, wherein one or more table entries 
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is in the form of an algorithm requiring data from the device reading the table to 
enable determination of the latency of the device holding said table. 

6. A system as claimed in Claim 5, wherein the determination on the 
5 basis of the algorithm is implemented by the device reading the table, said 

device having downloaded the algorithm from the device holding the table. 

7. A system as claimed in Claim 5, wherein the determination on the 
basis of the algorithm is implemented by the device holding the table, the results 

10 of the implementation being transmitted via said interconnection to the device 
reading the table. 

8. A system as claimed in Claim 1, wherein all destination devices 
maintain a respective table. 

15 

9. Data processing apparatus comprising the technical features of a 
source device in a system as claimed in Claim 3 or Claim 4. 

10. Data processing apparatus as claimed in Claim 9, further 
20 comprising the technical features of a destination device in a system as claimed 

in any of Claims 1 to 8. 

11. A data processing system comprising a cluster of devices 
interconnected for the communication of data in streams substantially as 

25 hereinbefore described with reference to the accompanying drawings. 

12. A data processing apparatus for use as a source device in a 
system comprising a cluster of devices interconnected for the communication of 
data in streams substantially as hereinbefore described with reference to the 

30 accompanying drawings. 
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LATENCY HANDLING FOR INTERCONNECTED DEVICES 

5 

A data processing system comprises a cluster of devices (16) 
interconnected for the communication of data in streams, particularly digital 
audio and/or video data. One of the devices (10) is a source device for at least 
two data streams to be sent to one or more other devices (12, 14) as destination 

10 devices of the cluster. To enable synchronisation of the stream presentations by 
the destination devices, some or all of the devices (10, 12, 14) carry respective 
tables (11, 13, 15) identifying, for that device, an identifier for each type of data 
stream that the device can process together with the processing delay for that 
stream. The or each such table is accessible via the cluster connection (18) to 

15 whichever of the devices, at source, destination or in between for the signal, is 
handling application of the necessary offsets. 
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(Figure 1) 




STREAM TYPE: 


DELAY: 


MPEG I 


0.5 seconds 


MPEG2 


1 second 


DV( <30 Mb/s) 


0.2 seconds 


DV ( >= 30 Mb/s ) 


0.4 seconds 


MPEG 7 


(0. 1 * a*; +- (0.005 * f), where x is the bit rate in Mb/s. and/is the frame rate 
in frames/second. 
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